GROUP 3. 


PERFORMANCE EVALUATION AND ASSESSMENT 


Chairman, Captain A1 Frink 


LOFT provides a unique new learning experience and an 
opportunity to look at aspects of performance other types of 
training have not provided. Areas such as crew coordination, 
resource management, leadership, and so forth, can be readily 
evaluated in such a format. While individual performance is of 
the utmost importance, crew performance deserves equal emphasis. 
Therefore, we feel that these areas should be carefully observed 
by the instructors as an area for discussion in the same way 
that individual performance is observed. 

There is an apparent conflict inherent in the purpose 
versus the application of LOFT. To be effective, it must be 
accepted by the crew members, and administered by the 
instructors as pure training — learning through experience. To 
keep open minds, to benefit most from the experience, both in 
the doing and in the follow-on discussion, it is essential that 
it be entered into with a feeling of freedom, openness, and 
enthusiasm. Reserve or defensiveness because of concern for 
"failure" must not inhibit participation. 

Yet, operators are responsive to safety concerns. They are 
charged with the responsibility of continuing training for those 
who require it. Thus, there is no such thing as a "no jeopardy" 
training exercise. Yet, it is essential to create that 
atmosphere. 

To a considerable extent, this conflict can be offset by 
the manner in which the instructor sets the scene during the 
pre-flight briefing. He should emphasize; 

o it is a pure learning experience; 

o it is a new training concept designed to accent 
crew command, coordination, communication, and 
full resource management; 

o he should emphasize the instructor's role, that 
he will not interfere regardless of developments; 

o that apparent mistakes may be made, but the crew 
should carry on — there is no one book solution to 
a LOFT exercise; 

o that there will be an opportunity for a full 
self-analysis during the debriefing; 
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o and that, he himself, will take notes and assist 
in the debriefing. 

To a certain extent, the instructors may have to be trained 
in areas such as resource management themselves, so that they 
are more closely attuned to those issues. However, at present, 
these standards are difficult to set and will hopefully evolve 
as more and more experience is gained with LOFT and resource 
management training . 

The instructor, because of the nature of LOFT, fulfills a 
very different role than in more traditional types of training. 
He is not an instructor in the traditional sense. For example, 
realism considerations dictate that the instructor not intervene 
or intrude in any way into the LOFT scenario. Thus, for purposes 
of the debriefing, it is crucial that the instructor serve 
primarily as a moderator. 

Instructors must have time to ‘ observe performance 
adequately. They should make detailed notes of observations 
made during LOFT so that they can guide the debriefing 
appropriately. LOFT places rigid demands on the instructor, 
handling ATC, running the scenario, and so forth. Thus, we 
recommend a tightly-scripted LOFT, and if possible, to have two 
instructors for three- man crews (one for two-man crews) so that 
performance can be adequately monitored. 

In the experience of companies who have utilized LOFT, it 
is often the case that crews tend to debrief themselves. Self- 
criticism and self-examination are almost always present in 
these situations and perhaps are much more effective than 
instructor criticism. In fact, crews are often much harder on 
themselves than the instructor would ever consider being. Thus, 
the instructor should do everything possible to foster this sort 
of self- analysis. 

In his role as moderator, the instructor can guide the 
discussion to points that he has noted need attention. 
Questions about certain procedures, mistakes, and so forth, 
should be asked whenever possible, and unless absolutely 
necessary, the instructor should avoid "lectures" about what is 
right and what is wrong. Obviously, the instructor should avoid 
the embarrassment of crew members as much as possible. A 
suggested format for the debriefing would include; 

o a positive general statement opening the 
discussion; 

o crew members should then be encouraged to discuss 
the operation as a whole and in part; 
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o in referring to his notes, the instructor must 
assure coverage of all aspects of the flight, not 
permitting one feature to dominate the 

debriefing ; 

o the instructor should mention (as appropriate) 
possible alternatives, better ways of 
accomplishing the objectives; 

o he should use questions to each member to further 
develop discussions, such as, "what if you had 
done. . . ; " 

o at the appropriate time, the instructor should 
summarize the debriefing. 

With respect to evaluation and assessment, everything 
should be done to assure crews participating in LOFT that their 
jobs are not in jeopardy every time they enter the simulator for 
a LOFT session. We feel that while "satisfactory completion" is 
an inescapable aspect of LOFT, at the same time it is hard to 
imagine "unsatisfactory training." In some cases, LOFT may 
underscore areas which need extra attention, but often, even 
serious mistakes made during LOFT are obvious and need no 
further attention. Even a session which results in a "crash" may 
be a "satisfactorily completed" LOFT if the learning provided by 
the experience cannot be improved upon. However, in some cases, 
mistakes may indicate deficiencies that need additional work. 
The way that this is conveyed to a crew member is of vital 
importance and represents a challenge to the companies and their 
instructors . 

During debriefing, both total crew performance and 
individual performances should be openly discussed and assessed 
by the instructor. Critical assessment of an individual must be 
mentioned in the presence of the full crew, but remedial details 
should be handled privately. Tact is required to avoid the 
appearance of satisfactory/unsatisfactory concepts. 

LOFT is, first and foremost, a learning experience. This 
committee feels that the success and acceptance of a LOFT 
program depends in great measure on the planning and preparation 
for the program. Scenarios must accent realism. Instructors 
should be carefully selected and trained in the art of briefing, 
conducting the program, and debriefing. 

Additional training for crew members, when indicated, must 
be handled in a low-key, non- threatening manner. If these 
factors are carefully handled, our committee feels that the 
evaluation/assessment chore will not necessarily detract from 
the pure training atmosphere, and will result in full 
acceptance . 
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Discussion 


CAPTAIN CAVANAGH: The one area in which I am somewhat 
disturbed, Al, is the categoric reference to fact that there 
must be two instructors present. I do not frankly think that 
there has been enough research or enough work done at this stage 
to say categorically, two instructors must be present. I think 
that it is important that instructors be qualified and properly 
prepared to observe the performance of all crew members present. 
I will grant that if there are two present that it is perhaps 
easier than with one, but I do not think it would be appropriate 
at this point to say there must be two. Automatic management of 
scenarios, automatic introduction of malfunctions, and various 
schemes might be used to minimize instructor workload; and might 
permit one to do a better job than two instructors could do in a 
situation where some of these aids are not present. I would 
personally rather see a report and recommendations that 
ultimately recognized these as legitimate concerns and issues 
that should be resolved in some other way. 

CAPTAIN FRINK; Dale, the committee discussed this at great 
length. The committee was unanimous in our resolve to find a 
way of presenting LOFT so that we could overcome the heart of 
the problem of acceptance of LOFT as pure training. We felt 
that the instructor's role was exceptionally important in this 
type of training versus other types of training that we do. He 
has a great deal of work to do. He has to do realistic 
communications, he has to monitor everything that goes on, to 
make sure that his work as conductor of the program is right on 
target; and to maintain the realism that is required. If you 
have a three-man crew with problems that are involving the 
engineer's panel, as well as the panels up front, and the need 
to monitor the specific actions of each crew member in addition 
to taking notes — we felt that note-taking was extremely 
important for the purposes of the debriefing — you have a 
situation involving a lot of work. After all these points were 
discussed, the committee recommended — nothing more — that if 
three people are part of a LOFT program, it would be better 
handled with two observing rather than one. 

CAPTAIN CAVANAGH; I recognize that it is a debatable issue. I 
submit that we might all look at this in terms of the three 
versus two-man crew concept. 

DR. LAUBER; I think that we will be getting back to this 
question once again when we get to the fourth working group 
report, instructor training and qualifications. That was one of 
the issues we put to them. I might say now that I hope during 
the course of the deliberations on that question, that the 
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conditions under which one instructor might be acceptable are 
discussed . 

CAPTAIN FRINK: Our group did consider some other matters. We 

felt that the terminology questions still poses a problem here. 
Yesterday, we stated that the acronym LOFT should only be 
applied to recurrent training. However, as we tried to talk 
about the line environment in other aspects of training, we 
found we were always talking about LOFT. No matter how you try 
to get around it, all types of training that utilize line 
conditions are referred to as LOFT even though they come nowhere 
near meeting what we have under 120-35, as capital L-O-F-T. 
Nevertheless, line-oriented flight training is a very common 
thing, and it applies to many versions of line-oriented 

training. We, therefore, felt that we would be better served if 
we made LOFT an umbrella term, and we are recommending that. 
Further, we should preface LOFT by the specific use that is 
being applied-- Recurrent LOFT, Transition LOFT, Upgrade LOFT, 
Remedial LOFT, etc. LOFT, itself, is in such general use that 
the term itself cannot be eliminated. It is going to be used in 
these other ways no matter how much we try to stop it. We 
cannot stop it by having it apply only to recurrent training. 
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